Lazy Sparse Stochastic Gradient Descent for Regularized Mutlinomial Logistic Regression

Stochastic gradient descent efficiently estimates maximum likelihood logistic regression coefficients from sparse input data. Regularization with respect to a prior coefficient distribution destroys the sparsity of the gradient evaluated at a single example. Sparsity is restored by lazily shrinking a coefficient along the cumulative gradient of the prior just before the coefficient is needed. 1 Multinomial Logistic Model A multinomial logistic model classifies d-dimensional real-valued input vectors x ∈ R into one of k outcomes c ∈ {0, . . . , k − 1} using k − 1 parameter vectors β0, . . . , βk−2 ∈ R: p(c | x, β) =  exp(βc · x) Zx if c < k − 1 1 Zx if c = k − 1 (1) where the linear predictor is inner product: βc · x = ∑